1. The Field of the Invention
This invention relates to cinematography and, more particularly, to novel systems and methods for virtual set creation and filming.
2. The Background Art
Cinematography involves the lighting, cameras, actors, props and other components required to film or photograph in moving format a story set in an environment, known as a set. Lighting is very important, actors likewise. Meanwhile, backgrounds may be created by filming on location. Similarly, backgrounds may be created by building a miniaturized set and taking great pains to photograph that miniature set and create an artificial background. Similarly, other mechanisms for filming an environment, such as front or rear projection, wherein a background “plate” of a scene is shot with one camera and later on that background is projected onto a screen in front of which actors are acting. This is another traditional mechanism for creating a background at a different time and a different place from the action that is photographed involving actors.
Nevertheless, such technologies cannot easily accommodate the parallax shift, the angle change, or the apparent angle change that is inherent in actual movement of a camera past a scene. To solve such parallax problems with screen projection technologies, computer controlled mechanical motion-capture jib devices must be used originally to film background plates, and then those same motion-capture devices must be later used within a soundstage, when the background plate is projected, to simultaneously film the foreground actors and other elements with precisely the same camera movements with which the plates were originally shot. This process can be very expensive.
Accordingly, in more recent decades, “green screen” technology has been used. This is most commonly observed on television when a weather person stands in front of an apparent screen, pointing or demonstrating with a hand toward the screen, but all the time looking at an orthogonal direction where that weather reporter is actually looking at the image that the television viewer is seeing. In other words, the weather person must watch himself on TV to know where he is in respect to the map because the map does not exist. That is, the map is virtual, and the weather person is actually standing in front of a green screen that is being edited out by the manipulation of the image and replaced by the weather map.
Green screen cinematography is used extensively in film making to place people within virtual environments. By filming actors in front of a green screen, post processing, or realtime processing, may rely on images taken of a background scene with a camera that is moving. Thus, that camera may actually move past, for example, a building, first seeing one side, then passing to front, then seeing the other side, and finally seeing the building recede in the peripheral vision of the user and of the camera. Thus, if the green-screen filming of actors and other foreground objects replicates this camera movement precisely, using motion-controlled hardware and software, this tends to solve the parallax problem. More recently, virtual 3D models have been created and linked to virtual cameras within 3D modeling software programs like Maya or Blender, to be combined with real-world camera recordings of actors and other foreground subjects, using match-moving software, to “sell” a shot with parallax. Nevertheless, green screens still leave many existing problems.
For example, shallow depth of field is critical in closeup shots, and often artistically preferable in many others, such as medium shots, or over-the-shoulder shots. That is, as a combination of the focal length, the proximity, the aperture, and other camera parameters, the depth of field becomes very short. Thus, actors may barely be within the total depth of field, and often actors would not be completely in focus for a shot. Thus, backgrounds would naturally be blurred. A 3D modeled scene that has been shot separately from the green-screen action of the actors will typically not have a proper depth of field illustrated in the composited shot without expensive, post-production image manipulations.
Likewise, green screen technology depends on “edge finding” algorithms, which try to recognize the edges of an object to be maintained or kept and other elements that are to be removed from an image. For example, the green screen may be removed by color. Meanwhile the edges of a hat, a hairdo, or the like may actually not be in complete focus or may not be easily detected as edges. Accordingly, even with the top green-screen removing technologies, hat, hair, or head images may be shrunk, resulting in more of an image removed than should have been, and other artifacts become introduced that tend to destroy the integrity of an image. This is particularly problematic in situations of focus shifting, or racking, between subjects, which is a very common tool in a cinematographer's toolbox.
Meanwhile, differentials in lighting cause additional problems with green screen technology. For example, if one part of an image is comparatively darker and has less lighting, intentionally or unintentionally, that artifact may be lost. If a green screen behind a blurred image is darker than other portions of the green screen, a clean “matting” out of the green becomes virtually impossible. Where an actor who is principal in a scene or a shot is to be better lighted than other actors and elements in the background, those background elements may be lost with green screen technology because the edge finding technology and the separation technology to separate the green screen from the images desired may completely eliminate or partially eliminate those elements.
Meanwhile, green screen technology itself introduces its own artifacts. For example, if lighting is not exactly even, then detection of regions by brightness or color may be difficult. Similarly, the green screen technology typically relies on match moving technology: trigonometric analysis of the image taken in order to calculate the location of a camera in 3D space during the shoot. Thus, if a camera is actually moving, then the green screen must be marked with markers that can be clearly seen and can be clearly detected and removed when replaced with the background images. This is not easily done if focal length or focus changes, if any part of the green screen is not perfectly in focus, if the lighting is not perfectly even along the green screen, and so forth. Meanwhile, any part of an actor or an element of a scene being shot if out of focus may be treated as if it is part of the green screen and removed from the image, or render the computation of the camera's precise location in 3D space impossible without more expensive, or manually intensive solutions.
Thus, in general, many subjects are not easily shot with virtual backgrounds or with green screen technology. Closeup shots, which require an artistically shallow depth of field, shots with shifting focus, shots in which the camera itself is moving, and shots with low lighting, and the like, become very difficult and expensive to shoot with pre-shot plates, or with virtual 3D model sets. And thus physical sets are typically built, or actually locations are used for shooting, with the specific purpose of shooting a single shot, a few shots, or many shots. But shooting within physical locations and with constructed sets within sound stages, can be very expensive.
Thus, in summary, green screen and virtual set technologies become very difficult to use and require much processing of the imaging in order to create credible views. Many common shots are not possible with the technology. And likewise, the building of full sized sets is time consuming, expensive, and is doubly expensive because of the need to maintain full crews of support, as well as full crews of actors while shooting real actors before real props before real sets.